Field of the Invention
The present invention generally relates to natural language processing, and more particularly to a method of establishing ground truths for a cognitive system.
Description of the Related Art
As interactions between users and computer systems become more complex, it becomes increasingly important to provide a more intuitive interface for a user to issue commands and queries to a computer system. As part of this effort, many systems employ some form of natural language processing. Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. Many challenges in NLP involve natural language understanding, that is, enabling computers to derive meaning from human or natural language input, and others involve natural language generation allowing computers to respond in a manner familiar to a user. For example, a non-technical person may enter a natural language query in an Internet search engine, and the search engine intelligence can provide a natural language response which the user can hopefully understand.
Different technologies can converge to provide resources for NLP, such as speech-to-text conversion (voice recognition). A user can say something to a computer system or cellphone, and the voice signal captured by the microphone is analyzed according to a particular human language or dialect to produce a text input or query in a computer-readable form. Text analysis is known in the art pertaining to NLP and typically uses a text annotator program to search text and analyze it relative to a defined set of tags. The text annotator can generate linguistic annotations within the document to tag concepts and entities that might be buried in the text. A cognitive system can then use a set of linguistic, statistical and machine-learning techniques to analyze the annotated text, and extract key business information such as person, location, organization, and particular objects (e.g., vehicles), or identify positive and negative sentiment.
A cognitive system (sometimes referred to as deep learning, deep thought, or deep question answering) is a form of artificial intelligence that uses machine learning and problem solving. Cognitive systems often employ neural networks although alternative designs exist. A modern implementation of artificial intelligence is the IBM Watson™ cognitive technology, which applies advanced natural language processing, information retrieval, knowledge representation, automated reasoning, and machine learning technologies to the field of open domain question answering. Such cognitive systems can rely on existing documents (corpora) and analyze them in various ways in order to extract answers relevant to a query, such as person, location, organization, and particular objects, or identify positive and negative sentiment. Different techniques can be used to analyze natural language, identify sources, find and generate hypotheses, find and score evidence, and merge and rank hypotheses. Models for scoring and ranking the answer can be trained on the basis of large sets of question (input) and answer (output) pairs. The more algorithms that find the same answer independently, the more likely that answer is correct, resulting in an overall score or confidence level.
Cognitive systems rely on ground truth to carry out their analyses. Ground truth is typically paired data, i.e., a sample input and a response, such as a question and an answer. Training data sets can be provided for ground truth, usually with subject matter experts weighing in on which training data is reliable. Curating high-quality ground truth is an important but difficult part of training a cognitive system. Existing approaches include using a brainstorming session to generate what the programmer thinks is representative training data, gamifying ground truth generation (by providing points/badges for creating x amount of ground truth), letting the users decide what kind of ground truth they will generate, or dictating what kind of ground truth the users will create, most likely by starting at low-accuracy components.